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SYSTEMS AND METHODS FOR THE MULTISPECTRAL 
IMAGING AND CHARACTERIZATION OF SKIN TISSUE 

This application claims the benefit of U.S. Provisional Application Nos. 

60/039,218 and 60/039,407, both of which were filed on February 28, 1997, and 

are incorporated by reference, herein. 

FIELD OF THE INVENTION 

This invention relates to methods and systems for the computer 

controlled analysis of digital images of skin tissue at a plurality of wavelengths, 

which may include those outside of the red-green-blue bands. The methods and 

systems further include the automatic characterization of the condition of the skin 

tissue, based on automatically computed values of parameters which are functions 

of characteristics of the skin tissue, based on the digital images. Skin lesions can 

be analyzed for determining whether the lesion is a melanoma, for example. 

Systems for digitally imaging and analyzing skin tissue are disclosed, as well. 

BACKGROUND OF THE INVENTION 

Melanoma is a usually fatal skin cancer, unless it is detected and 

surgically removed in its earliest stages. Early detection of malignant melanoma is 

difficult because early melanomas, those having a Breslow thickness less than 1 

mm, share many diagnostic features with benign lesions, such as dysplastic nevi 

or atypical melanocytic nevi. 

To aid in the analysis of lesions, conventional photography, referred 

to as "clinical imaging", has been used to image the lesion for further study. The 

effectiveness of clinical imaging can be compromised, however, by specular 

reflection by the skin. Polarizers have been used for polarized imaging, which 

minimizes specular reflection. 

Dermoscopy is another technique for examining skin, in which 

specular reflection is minimized. Dermoscopy also assists in clinically 

differentiating melanoma from its benign simulants by enabling the observation of 

features of pigmented melanocytic lesions that are not discernible by the naked 

eye. In dermoscopy, the skin is made more transparent to light by providing an oil 

layer over the skin, in front of the optical system. A glass plate is placed over the 

oil layer. The oil has an index of refraction between the index of refraction of the 
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horny layer of the skin and the glass plate. Standard magnifying optics may be 
used to enlarge the structures rendered visible on and under the surface of the 
skin by the oil layer. The region of interest can then be examined visually. Slides 
of the region of interest can be made, as well, for future study. 

Despite their similarities, most malignant melanomas differ in certain 
of their characteristics from other melanocyte lesions. A major advance in 
characterizing skin lesions based on certain of the observable differences between 
malignant and other lesions is the "ABCD" rule, where A=asymmetry, B=border 
irregularity, C=color variability, and D=diameter greater than 6 mm. A 
corresponding ABCD rule, where n D" refers to dermoscopic structures, such as 
brown globules, black dots or pigment networks within the lesion, is applied to 
dermoscopic images. Because the clinical and dermoscopic applications of these 
rules are subjective, they are not very reliable. 

When skin is illuminated by light, the light can be re-emitted by 
reflection, scattering or fluorescence. It is known in the art that the re-emission of 
light absorbed at different wavelengths by a region of interest of skin can provide 
different information. For example, as the wavelength of the light increases, its 
depth of penetration into the skin or other tissue also increases. Chromophores at 
different depths in the tissue therefore absorb and re-emit light at various 
wavelengths. Melanin and hemoglobin are examples of such chromophores. 
In addition, since the unaided eye cannot perceive light outside of the visible 
region or low-contrast structure in visible-light images, information which may be 
useful in diagnosing a lesion may not be directly observable. Digital acquisition 
and processing of dermoscopic images may, therefore, improve diagnostic 
reliability by employing more of the information residing in such images that is not 
directly observable. There have therefore been attempts to use objective, 
computer-based, image analysis algorithms that can discern meaningful 
differences between benign and malignant melanocyte lesions with sufficient 
accuracy. 

Computer processing of images requires that the image be in digital 
form. A digital image is an array of digital signals whose values are a function of 
certain characteristics of the subject of the image. When imaging skin lesions, the 
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digital images comprise digital signals whose values are a function of the re- 
emission characteristics of the skin and lesion, at the spectral bands of the light 
illuminating the skin. The array of numbers is obtained by spatial sampling and 
quantizing the intensity of images obtained with film or directly by electronic 
5 cameras. Practical limitations on the number of picture elements or pixels per unit 
area of image determine the achievable spatial resolution of the digital image. 
The digital image typically needs to be segmented to separate the digital signals 
which are a function of the skin lesion from the digital signals which are a function 
of the surrounding skin. 
10 Computer aided analysis has also been used to classify skin lesions 

using quantitative values indicative of particular characteristics of lesions, referred 
to as parameters. Based on histopathological diagnosis of lesions, algorithms 
have been developed which use linear or non-linear classifiers to combine 
parameters provided by the operator of an imaging device or a physician or 
15 computed by a processor, to yield a value which can be used to classify the lesion. 
Because some of the steps in the computer-aided analysis of which we are aware 
to date depend on subjective judgments of an individual, such analysis may 
provide highly variable results. 

The images heretofore available have been obtained with 
20 commercially available red-green-blue color imaging apparatus. Color 

photographic transparencies of skin lesions have been digitized and skin lesions 
have been directly imaged with "three-chip" digitizing cameras. Such cameras 
employ broad-band filter bandpasses that are ultimately based on the wavelength 
response of the human visual system and have large regions of overlap. 
25 Electronic images may also be obtained in narrower, non-overlapping 

filter bandpasses, which may reveal additional, wavelength-dependent differences 
between the images of melanomas and of benign lesions. However, such devices 
have had poor resolution and/or poor signal-to-noise characteristics which prevent 
the acquisition of digital images of melanocytic skin lesions of sufficient quality for 
30 effective application of machine vision techniques for lesion diagnosis. 

Existing imaging systems and processes also tend to suffer from an inability to 
provide the required repeatability of the values of extracted lesion parameters, due 
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in part to a lack of standardization with respect to spatially varying artifacts. The 
parameters, therefore, lack invariance to lighting and image exposure conditions, 
for example. Obtaining high signal-to-noise ratios in images recorded in narrow 
filter bandpasses, when exposure times are sufficiently short that the skin is 
effectively "frozen" during the exposure sequence, has also been difficult. In 
addition, since the optimum wavelengths for automatic characterization may not be 
the optimum wavelengths for visual observation, it may be difficult to reconstruct 
high-fidelity color images from the digital images for visual interpretation by a 
clinician. 

The assessment of wounds and burns through the appearance of 
color images present similar challenges. Existing technology for the imaging of 
skin in vivo for these purposes is also inadequate. Practical solutions to the 
problems of employing multispectral digital imaging of skin for the analysis of 
lesions, wounds, or other conditions have not been found. 

SUMMARY OF THE INVENTION 

The methods and systems of the present invention provide for the 
acquisition of digital images of skin at a plurality of spectral bands to automatically 
characterize the condition of the tissue based on the digital images. Spectral 
wavelength bands within and outside of the visible band may be used. In 
accordance with the present invention, a pigmented skin lesion can be 
characterized as malignant or benign, for example. The digital images comprise a 
plurality of digital signals whose values are functions of the condition of the tissue. 
The digital images acquired are subjected to objective and quantitative analysis by 
a digital processor to detect and identify abnormalities. The analysis includes 
image segmentation, parameter estimation and characterization of the skin. The 
estimation and characterization steps are automatic. The segmentation step may 
be automatic, as well. Subjective judgments are therefore minimized or 
eliminated. 

It has been found that generating the segmentation mask from a digital 
image acquired with light in a spectral band which does not penetrate deeply into 
the skin, such as a spectral band with a center less than about 500 nanometers, 
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provides superior results. After segmentation, estimated values which are 
functions of characteristics of the lesion, such as its texture, asymmetry, 
blotchiness, and border irregularities, are computed and used to automatically 
characterize the condition of the skin. Digital signals corresponding to hair or 
blob-like structures are preferably removed during segmentation. 

In accordance with the present invention, a method for characterizing 
the condition of a region of interest of the skin, wherein the absorption and 
scattering of light in different spectral bands by the region of interest is a function 
of the condition of the skin, is disclosed. The method comprises illuminating the 
region of interest of the skin by light in at least three spectral bands and digitally 
imaging the region of interest at the at least three spectral bands with the light re- 
emitted by the skin to generate digital images comprising digital signals whose 
values are a function of the condition of the skin. The digital images are provided 
to a processor which segments the digital images by generating a segmentation 
mask from a digital image in any one of the at least three spectral bands, 
computes at least one estimated value for each digital image at each spectral 
band which is a function of a characteristic of the region of interest within the 
segmentation mask, characterizes the condition of the skin based on the estimated 
values, and outputs the characterization of the condition of the skin. Preferably, 
the segmenting, estimating and characterizing steps are conducted without the 
intervention of an operator. Useful parameters include measures of the texture, 
asymmetry, blotchiness and border irregularity of the portion of the region of 
interest. 

The digital images may be obtained by directly imaging the region of 
interest with a digital camera, or digitally imaging color slides of the region of 
interest, through appropriately filtered light. 

The characterizing step may include comparing a weighted 
combination of the parameter values against a threshold value. The weight 
coefficients for c jch parameter value and the threshold value may be selected 
based on a training set of images of lesions or other skin conditions, whose 
condition has been determined, preferably through histological examination by a 
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plurality of doctors. Preferably, specificity is maximized under the constraint of 
100% sensitivity to melanoma. 

In accordance with another aspect of the invention, a system for 
characterizing the condition of a region of interest of skin includes means for 
illuminating the region of interest with light in at least three spectral bands and a 
camera for acquiring digital images of the region of interest based on the light re- 
emitted from the illuminated region of interest at each of the spectral bands, the 
digital image cot uprises digital signals whose values are a function of the condition 
of the region of interest. A digital processor segments the digital images by 
generating a segmentation mask from a digital image in any one of the at least 
three spectral bands and computes at least one estimated value for each digital 
image at each spectral band which is a function of the texture of the portion of the 
region of interest within the segmentation mask. The processor characterizes the 
lesion based on the estimated value or values. The other parameters discussed 
above may be uo^d, as well. 

The camera may be a single-chip or multiple-chip charge-coupled 
device which defects light in a plurality of spectral bands between the near 
ultraviolet to neci infrared. The filter means may be a plurality of interference 
filters mounted ui: a wheel for stepping any filter into a position intercepting the 
light from the light source. Preferably, at least one of the spectral bands has a 
center which lies between about 400 and 500 nanometers, at least one of the 
spectral bands ;<us a center which lies between about 500 and 600 nanometers, 
and at least one ciher spectral band has a center which lies between about 750 
and 1000 nanoi ^ters. 

DESCRIPTION OF THE FIGURES 

; rig. : (a) is a schematic illustration of a method and system of imaging 
a region of i.iic. of skin in accordance with the present invention; 

Fig. ; (b) is a schematic illustration of a plurality of narrow spectral 
bandwidths v ic. may be used to illuminate the skin in the embodiment of Fig. 
1(a); 
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Fig. 1(c) is a schematic illustration of alternative methods and systems 
for digitizing and analyzing color photographic slides of a region of interest of skin; 

Fig. 2 is a schematic illustration of preferred illumination and imaging 
portions of a computer controlled imaging system for direct imaging of a lesion; 

Fig. 3(a) is a flow chart of a calibration procedure for use with the 
present invention; 

Fig. 3(b) is a flow chart of a method of processing images for 
classifying lesions as malignant or benign, in accordance with the present 
invention; 

Figs. 4(a) and 4(b) are histograms of a malignant melanoma and of an 
atypical melanocyte nevus, respectively, showing two peaks in each histogram; 
Figs. 5(a) and 5(b) are histograms of another malignant melanoma and another 
atypical melanocyte nevus, respectively, showing three or more peaks in each 
histogram; 

Figs. 6(a) and 6(d) are digital images in the blue spectral band of 
another malignant melanoma and another atypical melanocyte nevus, 
respectively; 

Figs. 6(b) and 6(e) are digital images of the images of Figs. 6(a) and 
6(d) respectively, after thresholding; 

Figs 6(c) and 6(f) are digital images of the images of Figs. 6(a) and 
6(d), respectively, after iterative thresholding; 

Figs. 7(a) and 7(d) are digital images in the blue spectral band of 
another malignant melanoma and another atypical melanocyte nevus; 

Figs. 7(b) and 7(e) are digital images of Figs. 7(a) and 7(d), 
respectively, resulting from iterative processing and showing dark blobs outside 
the lesion area; 

Fiys. 7(c) and 7(f) are digital image masks of Figs. 7(b) and 7(d), 
respectively, resulting from image cleaning; 

Fi s. 8(a) and 8(e) are digital images in the blue spectral band of 
another malignant melanoma and another atypical melanocyte nevus, 
respectively, showing hair; 



SUBSTITUTE SHEET (RULE 26) 



WO 98/37811 



PCT/US98/03826 



8 

Figs. 8(b) and 8(f) are reverse intensity contrast images of the lesions 
of Figs. 8(a) and 8(e), respectively; 

Figs. 8(c) and 8(g) are digital images resulting from an averaging 
process applied to the images of Figs. 8(a) and 8(b), to remove hair; 

Figs. 8(d) and 8(h) are binary lesion masks resulting from the 
segmentation of the images of Figs. 8(c) and 8(g), respectively; 

Fig. 9 is a spatial filter used to remove hair; 

Figs. 10(a) - 10(c) are segmented digital images in the blue, green 
and red spectral bands, of the malignant melanoma whose histogram is shown in 
Fig. 5(a); 

Figs. 10(d) - 10(f) are segmented digital images in the blue, green and 
red spectral bands, of an atypical melanocyte nevus whose histogram is shown in 
Fig. 5(b); 

Fig. 11 is a chart of lesion parameters and their associated diagnostic 
accuracy, sensitivity and specificity when used individually; 

Fig. 12 is a plot of linear classifier values versus lesion identification 
number, for 41 malignant melanomas and 104 atypical melanocyte nevi; 

Fig. 13 is a plot of linear classifier values versus lesion identification 
number for 24 superficial spreading melanomas and 16 melanomas in-situ; and 

Fig. 14 is a plot of lesion parameter versus Breslow thickness for 24 
superficial spreading melanomas. 

DESCRIPTION OF THE INVENTION 

Fig. 1(a) is a schematic illustration of a method and system 1 in 
accordance with the present invention, by which images of the skin 2 are acquired 
by a camera nearly simultaneously at a plurality of spectral bands, K i=1 ( 2,...M, 
that are preferably effectively non-overlapping, as shown schematically in FIG. 
1(b). The skin is illuminated by a source of white light 3, which is filtered by 
narrow passband filters 4. The filtered light is preferably conveyed to the skin 2 
through a fiberoptic illuminator 5. The light re-emitted by the illuminated skin is 
imaged by a low-noise, high-resolution monochrome camera 6, which is preferably 
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an electronic charge-coupled ("CCD") camera. Digital images output by the 
camera 6 are provided to a computer 12 for processing. 

The computer 12 includes a digital interface 12a, a memory 12b and a 
digital processor 12c. A display 19 is preferably provided as well. The computer 
12 includes an input to a digital interface 12a for receiving the digital images. A 
memory 12b stores the digital images, and the software controlling operation of 
the imaging system, the image processing, and the classification and 
characterization of the lesion. The digital processor 12c, under control of the 
software, performs the calculations. The computer 12 has an output connected to 
a display 19, which can display the processed images and the results of the 
classification and characterization procedures for each image. The computer 12 
also preferably has outputs connected to the source of light 3 and the camera 6, 
for controlling their illumination level and exposure times, respectively, as 
described below. 

The image processing, classification or characterization and other 
programs can be implemented on a personal computer, using a programming 
language, such as FORTRAN or C. The memory 12b which stores the software 
can be any convenient media readable by the computer 12, such as the hard drive 
of the computer, i uad only memory, random access memory with a battery backup, 
electrically programmed ROM, electrically erasable ROM, floppy disc, or CD ROM. 
Other suitable media may be used, as well. 

Whe,. L;& fiiter bandpasses have minimal overlap, as in FIG. 1(b), 
each monochrome*^ image will contain spectrally independent information. Such 
spectral separation <s oelieved to be useful for differential diagnosis of skin lesions 
that contain vary :. . t i amounts of melanin, and of hemoglobin in different oxidation 
states, for example Spectral separation is also believed to be useful in 
distinguishing gr ..i ia .ion of tissue and other structural details of wounds in 
various stages ol healing. One or more of the wavelength bands may lie outside 
the visible regioi ^;ch as in the near infrared and/or the near ultraviolet, as long 
as the wavelengii lz wuhin the response range of the combined optical system 
including the eit ..ro.uc camera 6. 
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In accordance with another aspect of the invention, the digital images 
of skin lesions can be derived from color slides of the lesions obtained by clinical 
imaging, dermoscopy, or polarization imaging. Fig. 1(c) is a schematic illustration 
of alternative approaches to the acquisition and digitization of images of skin 
lesions from color slides. A photo camera 1 3 produces 35-mm color slides of a 
region of the skin 14. The camera 13 can be a Dermaphot ® camera from Heine, 
Optotechnik Gmbh & Co. AG, Germany, for example. The slides are typically 
stored in an archive 15. The slides are subsequently reimaged by a monochrome 
camera 16, which may be a CCD camera, that photographs each slide as it is 
illuminated by white light that has passed through a sequence of bandpass filters 
17 to create a color filtered version of the image. The slides can be illuminated at 
broad or narrow blue (B), green (G) and red (R) wavelength bands, respectively. 
The broad wavelength bands may overlap somewhat. In one example, the blue 
wavelength band was about 400 nm ± 30 nm, the green wavelength band was 
about 550 nm + 30 nm, and the red wavelength band was about 700 nm + 30 nm. 

Each of the filtered representations is recorded by the monochrome 
camera 16, which provides the resulting digital images 18 to an input of the 
computer 12. If an electronic camera such as a CCD camera is not used, the slide 
images could be digitized by any available commercial digitizer including three 
channels, one for red, one for green and one for blue, as long as the pixel size in 
the lesion plane after digitization is less than about 60 micrometers ( u |im M ). 

An appropriate CCD camera 16 is available from Electrim, Inc., 
Princeton, N.J. The ca;.i3ra 16 has a photographic macro-lens, wherein 1#/2.8 and 
f=100 mm. Preferably, i,*e spatial resolution of the CCD camera 16 provides 
pixels having a size about 10-30 nm in the lesion plane. The CCD camera 16 from 
Electrim, Inc., has 753 a 488 pixels. The spatial resolution with such a camera is 
approximately 21 x 24 \ at the lesion plane. Digital images of lesions obtained 
with this imaging systerv were used to classify lesions as malignant or benign, and 
to characterize lesions :.s invasive or non-invasive, as described further, below. 
The Electrim, Inc., CCu ^amera 16 has rectangular pixels. A CCD camera with 
square pixels would sirr piify the calculating procedures. 
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Alternatively, a 3-chip CCD camera 20, indicated in phantom in Fig. 
1c ( may be used to reimage the slides of the region of interest. The CCD camera 
20 provides digitized images for subsequent analysis by the computer 12. Broad 
bandpass filters, which are part of the CCD camera 20, produce a representation 
5 of the lesion as a set of three narrowband images. The filters are typically in 
accordance with CIE Standard Observer, wherein the bandwidths are broad. 

Fig. 2 is a schematic illustration of the illumination and imaging 
portions of a preferred computer controlled imaging system 22 in accordance with 
the present invention, for imaging a region of interest of skin including a lesion. 

10 The electronic camera 23 may be a 10-bit monochromatic electronic CCD camera 
23, such as the Xillix Model 1400, available from Xillix Technologies Corp., 
Canada. The Xillix camera is equipped with wide band, low distortion foreoptics, 
such as the XJP 1.9/0501, available from Jos. Schneider Werke, Germany. The 
lower distortion fore optics and the camera minimize chromatic aberrations of the 

15 optical system over the "muitispectral" sequence of exposures, enabling 

registration of images with sub-pixel accuracy, over the entire field of view. 
To ensure repeatability of imaging conditions and minimize required intervention 
by the operator, it is preferred that the system be operated at a preset f/stop. For 
cameras such as the Xillix Model 1400, exposure times are preferably controlled 

20 by the computer 12 through an electromechanical shutter that can operate reliably 
between minimum and maximum exposure times t mm and Ua*. 

The imaging system provides low-noise, high-resolution digital images 
at high data transf r rates, with low distortion imaging over the entire range of 
wavelengths covered by the collection of filters. The Xillix camera, discussed 

25 above, has a resolution at the skin surface of about 20 microns per pixel. 

The CCD camera 23 is preferably contained in a hand-held unit, represented 
schematically as b~x 24. The illuminator source 25 is a tungsten-halogen lamp 
whose intensity is controlled by a light-stabilized power supply 26 whose setting is 
automatically adjusted by the computer 12. A 150 watt lamp, such as the Phillips 

30 EJA, available fro. .. Phillips Electronics North America Corporation, N.Y., may be 
used, for example. The output of the lamp 25 is white light. A narrowband filter 27 
is provided betwec n the source and an optical fiber 28. A plurality of narrowband 
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filters, each one corresponding to a different spectral wavelength band, are 
mounted on a filter wheel 29. Preferred filter bandwidths are listed in Table 1, 
below. The filter wheel 29, which is driven by a stepping motor 29a, advances 
each filter to its proper position between the lamp 25 and the optical fiber 28, and 
5 holds each filter in position for a sufficient period of time. The computer 12 

controls the motor 29a. More or fewer filters may be used. Appropriate lenses 14 
are provided between the lamp 25 and the filter 27, and between the filter 27 and 
the optical fibers 28, as well. One or more fiber illuminators are provided for 
conveying the light from the source to the lesion. Two such illuminators 30a, 30b 

10 are shown in Fig. 2 for simplicity. Although the fiber illuminator illustrated is a 

bifurcated pair, a ring illuminator which provides more nearly uniform illumination 
at the skin surface, is preferred. An angle of illumination of about 20° is also 
preferred. A Fostec Model A0603 ring illuminator, available from Fostec, Inc., 
N.Y., may be used, for example. 

15 The hand-held portion of the system 24 of Fig. 2, which includes the 

camera 23, may be mounted on a cantilevered arm (not shown) that can be locked 
into position. 

The digital signals making up each of the digital images output from 
the camera 23 are provided to the computer 12. The computer 12 conducts image 
20 processing procedures on the digital images to calibrate the images, and to 

objectively segment, estimate parameters, and classify the lesions based on the 
estimated parameters. Operator judgment is not required at any point in the 
process. 

Control is maintained by the computer 12 over source intensity, filter 
25 position, and such camera settings as shutter timing, through the digital interface 
12a. Key control parameters are empirically chosen on the basis of feedback from 
histograms of trial images. The intensity of the lamp 25 may be maintained at a 
stable value, commensurate with the 10-bit dynamic range of the camera 26, by 
monitoring a secondary light source, connected electrically in series with the 
30 primary light source 25. The light output from the secondary source is monitored 
by a light sensor that is optically isolated from light reflections associated with the 
primary source. Such reflections may be caused by the filters that are located on 
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the filter wheel, or from the housing of the primary light source. This method 
provides optical feedback which is sensitive to changes in light intensity caused by 
changes in lamp lead resistance, for example, while it is insensitive to the variable 
amounts of light reflected from the filters, for example. By means of a closed 
control loop, the optical feedback from the secondary source is used to maintain 
long-term constant light output from the primary source. 

The apparatus of Fig. 2 can be used for either clinical imaging of the 
skin, wherein the skin is imaged directly, dermoscopic imaging, wherein a layer of 
oil is provided over the skin and a layer of glass placed over the oil layer, or 
polarized imaging, where a polarizer 31 is added to minimize specular reflection 
as shown in Fig. 2. In dermoscopic imaging, the index-matching oil sufficiently 
reduces the specular reflection to avoid the need for a polarizer. 
Instead of being positioned between the light source 25 and the optical fiber 28, 
the narrow bandpass filters 27 may be placed between the skin and the CCD 
camera 23 to filter the light reflected, scattered and radiated from the skin 2. 
The front end of the system preferably consists of a flat glass plate (not shown) for 
being placed over the skin. Light pressure is applied through the glass, onto the 
skin, throughout the imaging process. This helps to stabilize the region of interest 
against unwanted motion which could blur an image or which could lead to spatial 
misregistration between images obtained in different filter bandpasses. 

The preferred filters 27 for lesion imaging with a tungsten-halogen 
white light source 25 have the center wavelengths X { and bandwidths (FWHM) 
listed in Table 1, for i=l,2,.. M M ( M=I0, wherein the bands are labeled by j=M = 
0,1,.. .M-l. Such filters are available, for example, from Intor, Inc., Tucson, AZ. In 
each band, the exposure time is preferably selected to avoid saturation of the 
detector elements of the CCD camera 23, as well as to maximize the linear 
dynamic range over which the image data are recorded. These exposure times 
should be constrained to be within limits Urn and t max which are related to the 
electromechanical design of the shutter, optical throughput of the camera 23, and 
avoidance of image blur associated with motion during the exposure sequence. 
Suitable values of and Uax could be 10 ms and 550 ms, respectively, for 
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example. The choice of center wavelength and FWHM for the filter channels, as 
well as the corresponding exposure times, should preferably also take into account 
the following considerations: 

(a) The center wavelength and FWHM for at least two channels 
should be chosen so that characteristic absorption lines can be differentiated, 
such as those associated with melanin and hemoglobin; 

(b) For a given set of center wavelengths, there are upper limits on 
the associated bandwidths if spectral independence of data in different channels is 
to be maintained, as illustrated in FIG. 1(a); 

(c) Bandpasses should be chosen in the red, green and blue portions 
of the spectrum which enable "true-color" reconstruction of skin images that are 
suitable for visualization by clinicians; 

(d) The need for high signal-to-noise ratio in each image sets practical 
lower limits on the product of exposure time and filter bandwidth, especially at 
short wavelengths, where detector response falls off and lesion reflectance is low; 
and 

(e) The total time taken to acquire the images in all filter bands is 
preferably less than about three minutes, to minimize patient discomfort and 
possible motion. 

Based on considerations (d) and (e) above, and also taking into 
account the varying spectral reflectances of skin of different colors, the exposure 
times in each filter channel are preferably adjustable, with settings based on the 
dynamic range achieved on an empirical basis, with trial images. In this manner, 
both dynamic range and signal-to-noise ratio can be maximized for each filter 
channel. The preferred method is to choose Uxpiby iteration, based on intensity 
histograms of images of the skin obtained with trial exposures at each wavelength 
band. The histograms are analyzed to determine the number of pixels at the 
saturation intensity level, l tt t = 2 b -1 (1023 for b=IO bits). The exposure time is 
decreased if the number of saturated pixels exceeds a predetermined amount, 
such as 0.01% of the total. Conversely, to maintain high signal-to-noise ratio, the 
exposure time is increased if a predetermined percentile in the histogram, 99.9%, 
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for example, is reached at less than a preset threshold, such as 99.5% of Ut. The 
iteration process typically converges after two or three trials. 

The preferred exposure times at each wavelength for imaging skin of 
different colors to classify melanomas are listed in Table 1, for the embodiment of 
5 Fig. 2 with 10 filters. It has been found that for the blue channel centered at 450 
nm, the optimal exposure time for dark skin is 273 ms, which is more than double 
the optimal 107 ms exposure time for light skin. On the other hand, in the near 
infrared channel centered at 780 nm, the exposure times listed are much shorter, 
between 24 and 35 ms, and vary relatively little with skin type. The optimal 

1 0 exposure time for dark skin in the deep blue channel at 430 nm is at U™ s 550 ms, 
due to the low skin reflectance and relatively low optical throughput of the system 
at this short wavelength. Even with an exposure time this long, therefore, the 
image is less than fully exposed. Greater throughput at this wavelength could be 
achieved, at the expense of poorer response in the infrared. 

15 In Table 1, the FWHM at 450 nm, is 100 nm, which is much broader 

than for other wavelengths. It has been found that where images are desired for 
visual analysis as well as computer processing, the broad wavelength band at 450 
nm more closely matches the blue response of the human eye and is therefore 
preferred. In addition, the broad wavelength band provides data at a higher 

20 signal-to-noise ratio. 

Table 1 appears below: 
Optimal Exposure Times (msec) vs. Skin Color 
Filter Center Filter 



25 


Number 
(i=i-l) 


Wavelength 
(nm) 


FWHM 
(nm) 


Very Light 
Skin 


Medium 
Skin 


Tan 
Skin 


Dark 
Skin 




0 


430 


60 


405.2 


436.5 


484.4 


550.0 




9 


450 


100 


106.8 


124.9 


156.1 


273.3 




1 


500 


40 


56.4 


62.9 


88.7 


130.7 


30 


2 


550 


10 


44.2 


50.4 


71.3 


92.9 




3 


600 


10 


19.1 


24.0 


29.6 


39.2 




8 


650 


10 


74.5 


92.3 


104.9 


132.0 




4 


700 


10 


71.6 


86.0 


98.0 


114.4 
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5 


780 


30 


25.8 


29.1 


34.9 


23.6 


6 


880 


50 


34.1 


38.6 


44.8 


46.1 


7 


950 


60 


161.1 


187.6 


205.8 


212.0 



Tables similar to Table 1 can be readily constructed based on 
experimental results for other applications, where other spectral bands may be 
better suited. For example, in the analysis of wound healing, the ability to 
distinguish oxygenated from deoxygenated blood would be desirable. In addition, 
the wavelengths and exposure times in Table 1 reflect a balance between the best 
results for subsequent analysis of the images by a computer, and the best results 
for visual observation of the images. If visual observation is not necessary, other 
wavelength bands and exposure times may be used. 

FIG. 3(a) describes how the systems and methods of the present 
invention provide for calibration of the recorded images. The calibration procedure 
permits 10-bit image data to be recorded over a large linear dynamic range in 
each spectral band, independent of skin type. The recorded images can also be 
standardized for diffuse spectral reflectance. Consistent measures of reflectance 
ratios in different spectral bands can therefore be obtained, despite variations in 
illumination pattern with wavelength, changes in the position of the illuminator, or 
aging of the lamp, for example. 

First, the effects of dark current and "fixed pattern noise" are removed 
in Step 1 . N images are recorded by the camera without illumination. Preferably 8 
such dark images are recorded. The average of these N dark images, l D is 
calculated and stored in the computer 12. 

Second, spatial inhomogeneities in the illumination and in the 
response associated with each CCD pixel are removed in Step 2. A sequence of 
N' images of an illuminated flat, diffuse reflectance standard, such as a white 
Spectralon® target (R>99%) recorded. As above, N 1 is preferably 8. The N' 
images are recorded at each wavelength band. To average over local 
inhomogeneities in the reflectance standard, the target is moved continuously 
during the integration time and between exposures. A small motor, such as a 
reciprocating motor, may be used. The integration time and/or lamp intensity are 
adjusted by the computer 12 at each wavelength band until negligibly few of the 
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pixels are at or just below an intensity level corresponding to saturation. These NT 
"flat-field" images are averaged to reduce the effect of spatial non-uniformities in 
the reflectance standard, as well as to improve the detection signal-to-noise ratio. 
The resulting averages are stored in the computer as l wi> where i=1,2,...M. 
5 Next, monochromatic "raw data" images of the skin, I*, are captured by the camera 
and digitally acquired by the computer 12 within each filter passband, i=l,2,...M. If 
dermoscopic imaging is used, where a thin layer of mineral oil is spread between 
the skin and a cover glass is fixed in position in front of the camera, each image of 
the skin preferably contains an image of a narrow strip of oil-free, diffusely 

10 reflecting gray material, held in place on the inside surface of the cover glass, and 
located along one edge of the field of view. The material may be cut out of a 
Kodak "18% gray" card. Dermoscopic imaging is preferred for melanocyte 
lesions. The alternative clinical imaging mode is preferred for the imaging of 
wounds and burns because contact with the wound or bum by a cover glass is not 

15 desired. Although FIG. 2 indicates a lesion present on the skin 2, it will be readily 
understood that the same method will apply when a wound or burn is present, 
instead. In the clinical imaging mode, it is preferable to reduce specular 
reflections by employing the polarizer 31 , as indicated in FIG. 2. 

In either the dermoscopic or clinical imaging techniques, a fourth step 

20 is preferably provided, in which the raw data is compensated for dark current and 
fixed pattern noise and then normalized to the flat-field images. The dark-field 
compensation is performed by subtracting the stored average dark image l 0 both 
from the flat-field image l wl and from the raw data image !» . The ratio of the results 
of these subtractions is then taken. This standardizes the dark-corrected raw data 

25 to the flat-field image, compensating for spatially varying illumination arid pixel-to- 
pixel response variations. After the ratio is taken, the result is standardized to the 
maximum level, 2 b -1 which equals 1023 where b = 10 in a 10-bit data 
representation. The normalization process thus converts the image of the skin 
and the gray strip into a standardized diffuse reflectance map, with the result 

30 preserving a large linear recording dynamic range. In FIG. 3(a), the dark-field 

corrected and flat-field-normalized images, also referred to as "flat-field-calibrated" 
images, are denoted as U. In any image, standardization to maximum level can be 
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reinterpreted directly in terms of equivalent diffuse reflectance on the basis of the 
average gray level over the image of the gray strip, <lgray strips and the 
measured average diffuse reflectance of the gray strip, which is approximately 0.2 
and varies in a known and repeatable manner with wavelength. 
5 Preferably, the average image intensity in the gray-strip region is also 

used to calculate weighting factors for combining three or more monochromatic 
images to provide "true-color" visualizations of lesion images on the computer 12 
and display 19. This is preferably accomplished in Step 5, where the user selects 
the spectral bands to be used in the color visualization. Step 5 can take place 

1 0 prior to the imaging session. Four bands are currently preferred for such 

visualization. Filter bands j=3 and 8, in a 3:2 ratio, for the red (R) channel, filter 
band j=2 for the green (G) channel, and filter band j=9 for the blue (B) channel, in 
Table 1. As indicated in Step 6 of FIG. 3(a), the relative weights applied to the 
R:G:B channels are preferably inversely proportional to <lgray strips the average 

1 5 intensity over the portion occupied by the gray strip area in each image. This 
procedure tends to reconstruct the hues and saturations in the original scene to 
within accuracy limits associated with response nonlinearities of the display 19. 
To minimize the effects of such nonlinearities with display monitors such as the 
Sony Model GDM-175E1 Multiscan monitor, for example, the viewer may prefer to 

20 adjust the maximum brightness in the image to correspond to the maximum image 
intensity level of the monitor. A linear transformation step, which can be readily 
accomplished by commercial software such as Adobe Photoshop, may be used. If 
the digital images are derived from photographic slides, as in the embodiment of 
Fig. 1(c), steps 5 and 6 are not necessary. 

25 As indicated by dashed lines in FIG. 3(a), either the normalized 

monochromatic images resulting from Step 4 or the color visualization provided 
from Step 6 can be displayed on the display 19. Any or all of the monochromatic 
raw images could be displayed as well. 

Fig. 3(b) is a flow chart of a preferred method of processing images 

30 according to the present invention for characterizing the condition of a region of 
interest of the skin of a subject which includes a skin lesion. A skin lesion is 
selected in Step 50. Digital images of the lesions illuminated by light filtered at the 
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desired wavelengths of \, -A. 4 .. M are digitally recorded in Steps 52, 54, 56 and 
57..., as described above. Each of these digital images is processed separately. 
In Step 58, the image taken at the lowest spectral band is used to create a mask 
for segmentation. A blue spectral band is preferred. At Steps 60, 62, 64, 65..., 
5 each of the images of the lesion that correspond to different wavelengths are 
segmented by means of the segmentation mask obtained at Step 58. Lesion 
parameters are computed from each of the segmented images, in Step 66. Lesion 
parameters found to be useful for classifying and characterizing the lesion and 
statistical methods for computing the estimated values of the parameters, are 

10 discussed further, below. The estimated values of the parameters are provided to 
a linear classifier in Step 68. The linear classifier employs a linearly weighted sum 
of the individual parameters to derive a value used to classify the lesion as 
malignant or benign. A non-linear classifier such as a Gaussian quadratic 
classifier or an artificial neural-net classifier, each employing a suitable defined 

15 merit function, may be used as well. In either case, the numerical value produced 
by the classifier is subjected to a threshold test at Step 100, such that if the test is 
passed, the lesion is suspected to be malignant melanoma. If the test is failed, the 
lesion is declared not to be melanoma. The lesion could also be characterized as 
invasive or non-invasive with a different classifier. 
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I. SEGMENTATION 

The segmentation algorithms will now be described. The function of the segmentation 
algorithms is to discriminate between the lesion and normal skin in the field-of-view of the 
imaging device. This is a complex function since not only is the lesion appearance highly 
variable but so is the appearance of healthy skin due, for example, to the presence of 
blotches, hair, wrinkles, etc. The automatic algorithm described here is based on the 
images in the blue spectral band, from about 400 nanometers (run) to 500 run. This 
spectral band was selected because melanin absorption increases rapidly as the wavelength 
decreases. While the use of ultraviolet radiation could be advantageous, since ultraviolet 
radiation is carcinogenic, only low doses can be used. 

Segmentation in blue consists of several automatic steps: 

Location of major peaks in the histogram 

First, the histogram of intensity levels in the whole image is determined. Then, given 
a sliding window with the range of (2J P + 1) intensity levels, the number of peaks N p in 
the histogram over that range is determined. If N, < 2, the range is decreased by two 
levels and if N p > 3, the range is increased by two levels and the process is repeated until 
N p = 2 or 3. For most of the images in the data base used in this study, there are two 
major peaks in the histogram. Examples of such histograms are shown in Fig. 4(a) for 
a malignant melanoma and in Fig. 4(b) for an atypical melanocyte nevus. The lesions 
correspond to the lower intensity peak, since it is darker than the surrounding skin due to 
strong absorption by melanin at 400 nm. However, some lesions are quite inhomogeneous, 
and the automatic procedure described can find 3 major peaks, as illustrated in Figs. 5(a) 
and 5(b). 

Location of the intensity threshold 

If two major peaks are found in the intensity histogram, then the threshold value I th 
is selected to be at the histogram minimum between these two peaks, as indicated in Figs. 
4(a) and 4(b). In the case of three peaks, it has been found that, if the middle peak is closer 
to the lowest intensity peak, the threshold value is at the minimum between the middle 
and the highest intensity peak. If the middle peak is closer to the highest intensity peak, 
then the threshold value is at the minimum between the middle and the lowest intensity 
peak, as shown in Figs. 5(a) and 5(b). 

Iterative thresholding of the image 

The next step in image segmentation is iterative thresholding of the images. Given the 
intensity threshold value, image thresholding has been typically accomplished as follows. 
The intensity I(x t y) of a pixel at location (i,y) is set to zero if it exceeds J lh , i.e., 
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Figs. 6(a) and 6(d) are examples of digital images of malignant melanoma and atypical 
melanocyte nevus in the blue spectral band, respectively. Figs. 6(b) and 6(e) are images 
resulting from the direct thresholding as in Eq. (1). As shown in Figs. 6(b) and 6(e), 
"holes" can appear within the lesion. Therefore, an iterative approach is preferably used. 
First, the intensity of pixels at the image edges is set to zero. Then as each iteration 
proceeds, the intensity /(x,y) of a pixel at location (x,y) is set to zero if it exceeds I th 
and at least one of its nearest neighbors has zero intensity, i.e., 

M *' yj \/(*,y), otherwise, ( 2 ) 

where 

N nn = min[/(i - 1, y), /(i + 1 , y), /(i, y _ 1), J( x , y + 1), ) . ( 3 ) 

This procedure is iterated until there are no pixels with J(x, y) > I ih and & nearest neighbor 
with zero intensity. Typically, only a few iterations are required to complete this step. The 
resulting images are shown in Figs. 6(c) and 6(f). 

Figs. 7(a) and 7(d) are other examples of digital images of malignant melanoma and 
atypical melanocytic nevus, respectively. Figs. 7(b) and 7(e) are images resulting from the 
iterative thresholding described above. Various dark blobs are seen in the images outside 
of the lesion area. These are removed in the following step. 

Image cleaning 

Some of the blobs in the thresholded images arise naturally due either to dark spots on 
the normal skin or to hair as in Fig. 7(b). Others are artifacts such as the film edge at the 
top of the nevus image in Fig. 7(e), or dark bands at the image edges from the slide mounts. 
These bands are removed by automatically testing for their presence and then setting the 
intensity of appropriate pixels to zero. The remaining blobs could also be removed by 
determining the overall number and size, i.e., number of pixels, of connected blobs, and 
then setting to zero the intensity of pixels belonging to the small ones. However, since the 
size of some lesions exceeds 100,000 pixels, this would be computationally very intensive. 
Therefore, in practice, this step is preferably carried out as follows. First, perimeter pixels 
for all blobs in the image are located. The number of such pixels is typically less than 
10,000. Then, each of these perimeter pixels is assigned to a unique blob and its size, the 
number of perimeter pixels in the blob, is determined. The intensities of pixels belonging 
to blobs of size less than 30% of the maximum size for that image are set to zero. This 
process is iterated until all the small blobs are removed. Typically less than 10 iterations 
are needed. The intensity of all the nonzero pixels is then set to 1. The resulting binary 
lesion mask has the following property: 

j.f.^-/ 1 ! & Pixel at (x, y) belongs to lesion ; 
By ,V; ~\0, otherwise. ( 4 ) 
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Lesion Asymmetry 
Asymmetry parameter 

First^ l f° n ""T!^ " baSCd ° n m ° mcats of the intensity distribution 

Firs^ the les 10 n onentat>on angle * i, used to locate the principal axes, whiL are just the 
symmetry axes for symmetric lesions. The angle $ is computed from 

tan 26 = 2 < (* ~*c)(y ~ y e ) > 

< (x - x e )2 > _ < (y _ ye)2 > . (7) 

where the lesion intensity centroid is at 

x c = < x > and y e = < y > . 

< /(*,y) > = E, /(*.yUz,(x,y) 

" x:,i: y / t (x, y )-' o) 

where ii(x, y) is the segmented lesion ima*e In order t« . 

oi asymmetry described here requires rotation of the image by an anrie * so th*t 
principal axes are parallel to the ima«; av« T„ f v- ■ • , . S that 

faU«ta, Mymnet ' y flcto „ Pn-cpJ-ax,. coordinate sy« em the 

A = E, J e + n, y) - f L (g c - n > y )| 

A, = E» En l-fr ( x, y c + n) - J £ (x. ^ - n \\ 

E,E f /L(x,y) — - (106) 

The asymmetry parameter, 

- 4 = X ' + A " (id 

than fo, LSStL^S^ to * 

binary asymmetry parameter 

ion ot i^q. (4), then the corresponding asymmetry parameter A 6tn is 
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the fraction of the lesion pixels which do not have a counterpart on the other side of the 
principal axis. Thus, when based on the binary intensity distribution, parameter At, n is & 
measure of the asymmetry of the geometric shape of the lesion. 

Lesion Blotchiness 

Visually, many early malignant melanomas appear blotchy. In xnultispectral images 
there may be darker and lighter regions or blotches of rather homogeneous intensity. In 
color images, in contrast, there may be regions of different colors. Therefore, it is of interest 
to quantify such blotchiness in order to differentiate malignant from benign lesions. 

Blotchiness Parameter Based on Spatial Intensity Distribution 

The lesion is divided into N t "topographic" regions. If J ma * and I m in are the maximum 
and minimum intensities in the lesion in some spectral band, respectively, then a pixel at 
(z,y) belongs to the nth region if 

/m.„ + (n - I) 7 "" ~ /,n,a < J t (*,y) < I min + r» /m " . (12) 

iV| N t 

For nth topographic region defined in Eq. (12), a distribution of distances of pixels in that 
region from the intensity centroid of the binary lesion mask 



y) = \Z(x* - i e ) 3 + (y« - y c ) 2 (13) 

is obtained and its mean value < d n > and variance Var(d„) are computed. The measure 
of lesion blotchiness based on spatial intensity distribution is 

This parameter can be evaluated in every spectral band. 
Blotchiness Parameter Based on Ccntroids 

The lesion is again divided into N t "topographic" regions as defined in Eq. (12). An 
intensity centroid (x c (n), y c (n)), defined in Eqs. (8) and (9), is then computed for each 
such region separately. The blotchiness parameter based on the centroid is defined as 

C = (Xmax — Xmin){Y m az — Yintn)/A| (15) 

where, for example, Xma* is the maximum value of z c (n), and At is the lesion area in 
pixels. This blotchiness parameter is also determined in each spectral band separately. 
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Blotchineaa Parameter Baaed on Spatial Color Distribution 

The "color" in this analysis is uot related to the visual perception of color. It is 
a quantitative descriptor of the relative intensities in red, blue, and green channels in a 
particular pixel. 

All the other lesion parameters described here involve analysis of images in each 
spectral band separately. Therefore, absolute calibration of image intensities was not 
necessary. However, in order to describe the color distribution, normalization of intensities 
in red, green, and blue spectral bands is needed, so that intensities in the three channels 
are equal for white. In the spherical color coordinate system, 

R (x v ) = J «(*»y) 

' y) /n(*,y) + /B(*,y) + /c(*, v )' 

G{X ' V) = /*(*,y) + ll{x\l] + / c (x,y) » ( 16 ) 
where the subscripts R, G, B refer to red, green, and blue spectral bands, are chosen as 
the independent variables. The lesion is then divided into color regions as follows. First 
R{x, y) and G(x, y) are divided into N R and N G topographic regions. A color region is 
defined as a particular combination of two topographic regions. The total number of color 
regions is 

N c = N R x N c . (17) 
The blotchiness parameter based on color is denned in analogy with Eq. (14): 



Lesion Border 

Border Irregularity Parameter 

Border irregularity is a well-known feature of malignant melanomas. It is typically 
defined as the ratio of the measured lesion perimeter to the perimeter of a circle with 
the same area as the lesion. Since perimeter is difficult to estimate reliably, a statistical 
descriptor of border irregularity is used here. In addition, many lesions are elongated and 
an ellipse is a better approximation for such lesions with regular borders than a circle. 

Using the binary lesion mask of Eq. (4), the lesion intensity centroid from Eq. (8), 
orientation angle from Eq. (7), area, and the aspect ratio denned as 



Aft v /< J/ ~ J « > 2 
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where primes refer to the coordinate system defined by the lesion principal axes, are 
determined. These values are then used to construct an ellipse that is the best regular 
approximation to the lesion border. For each lesion border pixel at (z», y k ), its angle with 
respect to the horizontal axis: 

and the location of the ellipse border for the same angle (z«(*), y,(^)) are determined. 
The distribution of distances between the ellipse border and lesion border: 

V6) = <*»(*», y»)-d e (*), ^21) 

where 

y*) = \/(i»-x e ) 2 + (y 6 -y e y (22) 

and 



<W) = y/'l+y 2 ., (23) 
is obtained and the border irregularity parameter is denned as 



<d»> ' (24) 



Border Gradient Parameter 

Another parameter that quantitatively characterizes lesion border is the measure of 
intensity gradients across the lesion borders over the length scale denned by n„ in units 
of pixels. For each lesion border pixel at (r t ,y 6 ) one determines whether pixels'at (z» ± 
"» • V» ± n >) ■« ** the border. If they are not, then the gradient is defined as 



1, 



G{z k ,y>) = + n„y) - J(z - n„y)\ + \I(x,y + n,) - J(x,y - n,)\) ; (25a) 
otherwise, if pixels at (z ± n,,y) are not on the border, 

C(z fc ,y t ) = |/(* + n„y)- J(*-n # ,y)|, (256) 
or, if pixels at (r.y ± n 9 ) are not on the border, 

G(x k ,y t ) = |/(z,y + n,)-/(r,y-n p )|. (25c ) 
The border gradient parameter is defined as 

G ~ <G> (26) 
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Lesion Texture 

The description of lesion texture is particularly vulnerable to subjective judgement. 
The quantitative evaluation of lesion texture parameters is possible only using computer- 
based image analysis. While many such parameters are possible, those found to be helpful 
in discriminating between malignant melanomas and atypical melanocytic nevi are de- 
scribed below. 



Texture Parameters Based on Local Intensity Variations 

Texture parameters are defined over a length scale r»i in units of pixels. For example, 
consider a pixel located at (x, y) in the lesion. Let Ii and I. be the minimum and the 
maximum intensities in an image in the 2n f + 1 x 2n t + 1 window around this pixel, i.e., 
in the range [x — n, t x + n f ] and [y - m,y + m]. Consider a variable 

Ci(x,y) = /m ^ 7 ' . (27) 
The first two texture parameters are defined as: 

< Ci > 



(28) 



and 

T2 = v/VarCd) (29) 
Another texture parameter uses the following variable: 

<2»(*. y) = 4u»(0, 0) + u>(-fi,, 0) + u;(n,,0) + u»(0, -n,) + u>(0, n t ) 

-2[u>(-n t ,-n t ) + w(-n tt n t ) + u>(n,,-n t ) + w(n t ,n,)] ' ^ 

where 

w(x\ J ) = J(* + y + j)/I(x, y) • (31) 
If the value of C 3 is negative, it is set to zero and the corresponding texture parameter is 

<C 3 > < 32 > 

Another variable that leads to a texture parameter useful for classification of melanocytic 
lesions is: 

C 4 (x, y) = 8ti>(0,0) - u.(-n,,0) - u>(n,,0) - u»(0, -n t ) - u»(0,n f ) 

(33) 

- u>(-n<, -n ( ) - w(-n t) nt) - w(n i% -n r ) - w(n tt n t ) 
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Again, if the value of the variable is negative it is set to zero and the corresponding texture 
parameter is 

y/Var(C<) 

" <C 4 > <»> 



T4 = 



Texture Parameter* Based on Ptpmented Network 

Texture parameters have also been developed by considering the properties of a pig- 
mented network. These texture parameters are measures of variability in the area of the 
dermal papillae and in the aspect ratio (length/width) of the rete ridges. 

Since dermal papillae appear as the brighter part of the network, one seeks all the 
local maxima over a 2n, + 1 x 2n« + 1 window. Starting from such a maximum at (x m , y m ), 
one finds local one-dimensional minima in eight directions (2 vertical, 2 horizontal, and 
4 diagonal) and locates the vertices of an octagonal region one pixel closer to the max- 
imum intensity pixel than the minimum pixel. Such octagonal regions approximate the 
areas of dermal papillae A dp which are computed from the known location of vertices; the 
corresponding texture parameter is 

Some of the areas determined by this algorithm are due to bubbles visible in some of these 
dermoscopic images. However, since there are typically on the order of hundreds of areas, 
and on the order of tens of bubbles, the statistical parameters should not be significantly 
biased by this artifact. 

The aspect ratio of rete ridges is determined in a similar fashion, although one starts 
with local minima since rete ridges appear dark in the images. The vertices of an octagonal 
region are determined in this case from one-dimensional maxima in the eight directions. 
The maximum and minimum extents of this region are then determined and the aspect 
ratio R is computed. This texture parameter then is 



T6 VVarTR) 



<~R>~ ■ (*) 
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HI. LESION CLASSIFICATION 

Selection of lesion parameters for classification was done by determining the maximum 
diagnostic accuracy for malignant melanoma for each parameter computed in every spectral 
band available for the training set of images. As mentioned above, diagnostic accuracy, 
sensitivity to malignant melanoma and specificity for the selected twenty two parameters 
are shown in Fig. 11. These parameters were then used as input to the linear classifier. 
Nonlinear classifiers may be used as well. 

For each lesion Jb the linear classifier is 

22 

= £ (37) 

where X n (k) are the parameters for the Jbth lesion and weights u>„ are to be determined so 
that a specified function F(L) attains ma nira urn value. The following functions F(L) were 
used: 1) specificity under constraint of 100% sensitivity to malignant melanoma for the 
training set which included 41 malignant melanomas and 104 atypical melanocyte nevi; 
(2) classification accuracy for the 24 invasive and 16 noninvasive malignant melanomas of 
the training set; and 3) correlation with the Breslow thickness for the 24 invasive malignant 
melanomas. 

Given any training set of lesion images and corresponding set of lesion image param- 
eters, the weights that maximize F(L) are found as follows. First, an initial range and 
resolution A for u>„ are selected. For each allowed set of values of w nt the values L(k) 
are computed for each lesion. The value F(L) is determined based on the input from 
histopathoiogical evaluation of the lesion based on a biopsy, such as the diagnosis of the 
lesion as benign or malignant, and the Breslow thickness for a malignant melanoma. The 
range of u> n 's is adjusted until the maximum value of F(L) is inside the range. Then the 
resolution A is reduced by a factor of two, and the process is repeated until A reaches 
specified minimum value A m , n - This procedure determines the weights w n only up to a 
multiplicative constant. It is noted that the classifiers resulting from a particular training 
set are applicable only to images with a specific spatial and spectral resolution, and that 
lesion images obtained with a different imaging system may require the development of 
different classifiers, by the procedures described above. 

Since detection of melanoma in its early stage significantly improves prognosis, there 
is a need for reliable methods of early detection. Clinical evaluation of melanocyte lesions 
is, however, a problem since reliable differentiation between early malignant melanoma 
with Breslow thickness less than 1 mm and atypical melanocyte nevus is difficult even 
for experienced dermatologists. In order to detect as many early melanomas as possible, 
weights in the l i ne ar classifier are preferably chosen to maximize specificity under the 
constraint of 100% sensitivity to malignant melanoma for the training set. For each set 
of weights, one finds the threshold value Lth of the linear classifier such that a Jbth lesion 
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is classified as suspicious of malignancy if L(k) > L lh , and as benign otherwise. The 
resulting classifier for the training set is 

Z-i = 0.025 A ti n + 0.090 A* + 0.069 A, + 0.160 A r + 0.128 C b 

+ 0.095 CI + 0.038 B + 0.107 Tl, + 0.064 T2 t + 0.018 T2 r (38) 
+ 0.111 T3 t + 0.167 T3, +0.268 T5 t 

where the weights are normalized so that the threshold value equals one. This classifier 
with sensitivity to malignant melanoma of 100% and specificity of 85% is shown in Fig. 
12. Statistical significance of the specificity and sensitivity was assessed by considering 
the binomial probabilities for the value of L\ to exceed the threshold for the 41 malignant 
melanomas and 104 atypical melanocyte nevi of the training set separately. At the 95% 
confidence level, one finds that sensitivity is not less than 93% while specificity is not less 
than 79%. Since there are several melanomas very close to the threshold value, a practical 
classifier may use a threshold value that is less than one. It has been found that this set of 
145 images is sufficient to yield statistically significant results. A greater number of images 
may be used, as well. 

Some of the noninvasive melanomas, called melanomas in-situ, are confined to epider- 
mis and are 100% curable by surgery. The invasive melanomas, i.e., superficial spreading 
melanomas in our data base, require more extensive surgery. Therefore, it is of clinical 
interest to differentiate between invasive and noninvasive melanomas and a linear classifier 
was trained to perform this task. This classifier, with weights chosen to maximize the over- 
all classification accuracy for the 24 superficial spreading melanomas and 16 melanomas 
in-situ of the training set is 

L 2 = -1.00 A» - 0.14 Bl, - 2.47 Bl r - 0.4 C» - 0.98 CI 

- 1.17 T2 r + 0.53 T4 fc + 1.98 T5» + 1.58 T6» - 0.73 (39) 

where a constant was subtracted from the classifier values to obtain the threshold value of 
zero. This classifier, with overall classification accuracy of 92.5%, is shown in Fig. 13. 

Since prognosis for invasive melanomas correlates strongly with the Breslow thickness 
a War function of lesion parameters Q was trained to maximize the Pearson correlation co-' 
efficient between Q and the Breslow thickness for a set 24 superficial spreading melanomas 
This function is 

Q » 0.955 A, - 1.391 A r + 2.791 Bl h - 1.320 Bl, + 0.146 C» 

+ 0.267 C r - 0.506 B + 0.202 Tl, + 1.476 T2 r - 0.485 (40) 

and is shown in Fig. 14. Even though there are only 24 superficial spreading melanomas in 
the data base, the high correlation of 0.77 is statistically very significant since p = 9 x 10" e . 
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The classifiers of Eqs. (38 H 40 ) applicable to the imaging system described above, 
with respect to Fig. 1(a) wherein the monochrome camera 16 was used to digitize slides. 

For other imaging systems, having different spatial and spectral resolution, different 
classifiers may need to be developed, based on a sufficient data base of lesion images 
obtained with that imaging system, in accordance with the procedures described above. 
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The segmentation, parameter estimation and classification programs 
described in Sections Mil, above, can be implemented on any personal computer, 
using a programming language, such as FORTRAN or C. The program can be 
stored on any convenient media readable by a computer, such as read only 
memory, ("ROM"), random access memory with a battery backup, electrically 
programmed ROM (EPROM), electrically erasable ROM (EEPROM), floppy disc, 
CD ROM, or a hard disc. Other suitable media may be used, as well. 

While the procedures of Sections Mil were described with respect to 
digital images obtained by imaging color photographic slides of skin lesions with a 
monochrome CCD camera 16 in accordance with the system of Fig. 1c f these 
procedures are readily adaptable to the analysis of digital images of skin lesions 
acquired directly from the region of interest of the skin with a monochrome CCD 
camera 6 of Fig. 1a and Fig. 2. 

In the process described in Sections Mil, above, segmentation was 
conducted in the blue wavelength band. The segmented mask in blue was then 
applied to images in the red and green wavelength bands. Where images at 
additional wavelengths are provided, segmentation is preferably conducted of the 
image at the shortest available spectral band where the contrast between the 
lesion and normal skin tends to be highest because of strong absorption by 
melanin. The segmented mask is then applied to the images in the remaining 
spectral bands. These steps are shown in Fig. 3b. 

In addition, where parameters are described in terms of red, green and 
blue wavelength bands in Section Mil, parameters can be derived at each of the 
other wavelengths used in the systems and methods above, in accordance with the 
procedures described in Section Mil. The additional parameters can be readily 
used to develop a classifier, also by the processes described in Section Mil. 

In addition, where parameters are described in terms of red, green and 
blue wavelength bands in Section Mil, parameters can be derived at each of the 
wavelengths used in the systems and methods above, in accordance with the 
procedures described in Section MM. The additional parameters can be readily 
used to develop a classifier, also by the processes described in Section Mil. 



SUBSTITUTE SHEET (RULE 26) 



WO 98/37811 



PCTYUS98/03826 



34 

The references cited above are incorporated by reference herein. 

While preferred systems and methods for practicing the present 
invention have been described above, it is understood that departures may be 
made from the systems and methods, without departing from the scope of the 
present invention, which is defined by the following claims. 
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We claim: 

1 . A method of characterizing a skin lesion wherein the 
absorption and scattering of light in different spectral bands by the skin lesion 
is a function of the condition of the skin, the method comprising: 

illuminating a portion of the skin including the region of interest 
by light in at least three spectral bands, one of which is a blue spectral band; 

digitally imaging a portion of the skin including the region of 
interest at the at least three spectral bands with the light re-emitted by the 
portion of the skin to generate digital images comprising digital signals whose 
values are a function of the condition of the region of interest of the skin; and 

providing the digital images to a processor, wherein the 

processor: 

segments the digital image by generating a segmentation mask 
defining the boundary of the region of interest from a digital image in the blue 
spectral band, without operator intervention; 

automatically computes at least one estimated value for each 
digital image at each spectral band which is a function of a characteristic of 
the portion of the region of interest determined by the segmentation mask, 
without operator intervention; 

characterizes the condition of the skin as malignant or benign 
based on the estimated values, without operator intervention; and 

outputs the characterization of the condition of the skin. 

2. The method of claim 1 , further comprising estimating at least 
one value which is a function of the texture of the region of interest. 

3. The method of claim 2, wherein the computing step 
comprises estimating values which are statistical measures of local intensity 
variation in the digital images in each spectral band, which are a function of 
the texture of the region of interest. 

4. The method of claim 2, wherein the computing step 
comprises estimating values based on the ratio of standard deviation of the 
areas of dermal papillae to their mean within the segmentation mask. 
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5. The method of claim 2, wherein the computing step 
comprises estimating values of the average and standard deviation of the 
thickness of rete ridges within the segmentation masks. 

6. The method of claim 1 , further comprising estimating at least 
one value which is a function of the asymmetry of the region of interest in 
each spectral band, for two principal axes of the segmented image by: 

locating the principal axes by computing an orientation angle; 
computing the intensity centroid; 

rotating the digital image such that the principal axes are 
parallel to the image axes; and 

estimating the asymmetry values for each principal axis based 
on the intensity centroid; and 

summing the estimated asymmetry value for the two principal 

axes. 

7. The method of claim 1, further comprising estimating at least 
one value which is a function of the blotchiness of the region of interest. 

8. The method of claim 1, further comprising estimating at least 
one value which is a function of the irregularity of the border of the region 

of interest by estimating a value which is a statistical measure of the 
deviation of the border of the segmentation mask from the border of an 
ellipse of the same area, aspect ratio, and orientation as the segmentation 
mask. 

9. The method of claim 1, further comprising estimating a value 
which is a function of the gradient at the border of the region of interest by 
estimating a statistical measure of the gradient values of the intensity of the 
digital images across the border of the segmented images, at each spectral 
band. 

10. The method of claim 1, further comprising characterizing 
the type of lesion as invasive or non-invasive. 

1 1 . The method of claim 1 , wherein the segmenting step 
comprises generating the segmentation mask from a digital image by: 
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removing digital signals from the digital image which correspond 
to hair structures; 

deriving a threshold from a multimodal histogram of intensity 

levels; 

iteratively applying the threshold to the digital signals of the 
digital image; and 

removing digital signals which correspond to small blob-like 
regions from the masked image. 

12. The method of claim 1, wherein the digital imaging step 
further comprises digitally imaging the region of interest with a digital camera. 

1 3. The method of claim 1 , further comprising: 
photographing the region of interest with a color camera to form 

color photographic slides; and 

illuminating the color photographic slides with light in each 
spectral band; 

wherein the digital imaging step comprises digitally imaging the 
illuminated color photographic slides of the region of interest with a digital 
camera. 

14. A method of characterizing the condition of a region of 
interest of skin, wherein the absorption and scattering of light in different 
spectral bands by the region of interest is a function of the condition of the 
skin, the method comprising: 

illuminating a portion of the skin including the region of interest 
by light in at least three spectral bands; 

digitally imaging the portion of the skin including the region of 
interest at the at least three spectral bands with the light re-emitted by the 
portion of the skin to generate digital images comprising digital signals whose 
values are a function of the condition of the region of interest of the skin; and 

providing the digital images to a processor, wherein the 

processor: 
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segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest from a digital image in 
any one of the at least three spectral bands; 

computes at least one estimated value for each digital image at 
each spectral band which is a function of a characteristic of the region of 
interest determined by the segmentation mask; 

characterizes the condition of the region of interest of the skin 
based on the estimated values; and 

outputs the characterization of the condition of the region of 
interest of the skin. 

15. The method of claim 14, wherein the estimating and 
characterizing steps are conducted without the intervention of an operator. 

16. The method of claim 14, wherein the segmenting step is 
conducted without the intervention of an operator. 

17. The method of claim 14, wherein the illuminating step 
further comprises illuminating the region of interest with light in at least one 
spectral band which penetrates to the papillary dermis and is re-emitted 
therefrom. 

18. The method of claim 17, wherein the digital imaging step 
further comprises digitally imaging the region of interest with a digital camera. 

19. The method of claim 17, wherein the illuminating step 
further comprises illuminating the region of interest with light in a near 
infrared spectral band. 

20. The method of claim 14, further comprising suppressing 
specular reflections prior to the digital imaging step. 

21 . The method of claim 20, wherein the processor converts 
the digital signals of each of the digital images into values corrected for non- 
uniformities of illumination and of response prior to the segmenting step. 

22. The method of claim 14, further comprising: 
photographing the region of interest with a color camera to form 

color photographic slides; and 
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illuminating the color photographic slides with light in each 
spectral band; 

wherein the digital imaging step comprises digitally imaging the 
illuminated color photographic slides of the region of interest with a digital 
camera. 

23. The method of claim 14, wherein the segmenting step 
further comprises applying the segmentation mask to the digital images in the 
other spectral bands. 

24. The method of claim 14, wherein the segmenting step 
comprises generating the mask at the shortest available wavelength. 

25. The method of claim 14, wherein the illuminating step 
comprises illuminating the region of interest by light in at least one spectral 
band whose center is between about 400 to about 500 nanometers, and the 
segmenting step comprises generating the mask from a digital image at the 
spectral band between about 400 to about 500 nanometers. 

26. The method of claim 16, wherein the segmenting step 
comprises generating the segmentation mask from a digital image by: 

removing digital signals from the digital image which correspond 
to hair structures; 

deriving a threshold from a multimodal histogram of intensity 

levels; 

iteratively applying the threshold to the digital signals of the 
digital image; and 

removing digital signals which correspond to small blob-like 
regions from the masked image. 

27. The method of claim 16, wherein the segmenting step 
comprises generating the segmentation mask by comparing estimated values 
which are a function of textures within the digital images with a threshold. 

28. The method of claim 27, further comprising generating the 
segmentation mask by comparing the estimated texture values to a threshold 
derived through statistical analysis of each digital image. 
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29. The method of claim 14, wherein the computing step 
comprises estimating at least one value which is a function of the texture of 
the region of interest determined by the segmentation mask. 

30. The method of claim 29, wherein the computing step further 
comprises estimating values which are a function of the texture of the region 
of interest determined by the segmentation mask separately in each spectral 
band, based on the same segmentation mask. 

31. The method of claim 29, wherein the computing step 
comprises estimating values which are statistical measures of local intensity 
variation in the digital images in each spectral band which are a function of 
texture. 

32. The method of claim 14, wherein the computing step further 
comprises estimating a value which is a function of the asymmetry of the 
segmented image in each spectral band, for two principal axes of the 
segmented image. 

33. The method of claim 32, wherein the computing step further 

comprises: 

locating the principal axes by computing an orientation angle; 
computing the intensity centroid; 

rotating the digital image such that the principal axes are 
parallel to the image axes; 

estimating asymmetry values for each principal axis based on 
the intensity centroid; and 

summing the estimated asymmetry values for the two principal 

axes. 

34. The method of claim 33, wherein the computing step further 
comprises computing the intensity moment with a binary intensity distribution. 

35. The method of claim 14, wherein the computing step further 
comprises estimating at least one value which is a function of the blotchiness 
of the segmented digital image, the estimated blotchiness value being 
defined through statistical properties of the spatial distribution of topographic 
regions of the digital images at each spectral band. 
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36. The method of claim 35, wherein the computing step further 
comprises determining the centroids of topographic regions of the segmented 
digital image at each spectral band. 

37. The method of claim 14, wherein the computing step 
comprises estimating a value which is a statistical measure of the deviation of 
the border of the region of interest from the border of an ellipse of the same 
area, aspect ratio, and orientation as the segmentation mask. 

38. The method of claim 14, wherein the computing step 
comprises estimating a statistical measure of the gradient values of the 
intensity of the digital images across the border of the segmented images, at 
each spectral band. 

39. The method of claim 14, wherein the computing step 
comprises estimating values based on the ratio of standard deviation of the 
areas of dermal papillae to their mean within the segmentation mask. 

40. The method of claim 14, wherein the computing step 
comprises estimating values of the average and standard deviation of the 
thickness of rete ridges within the segmentation mask. 

41. The method of claim 14, wherein the characterizing step 
comprises comparing a weighted combination of parameter values against a 
threshold value. 

42. The method of claim 41 , wherein the condition of the region 
of interest to be characterized is the presence of a melanoma and weight 
coefficients for each parameter value and the threshold value are selected to 
maximize specificity, under the constraint of 100% sensitivity to melanoma, 
on a representative set of training images 

43. The method of claim 14, further comprising calibrating each 
pixel location in the digital image in each spectral band with respect to stored 
images of a white target material having known diffuse reflectance, each of 
the stored images being an average of a plurality of images acquired at each 
spectral band, while the material undergoes continual in-plane motion. 

44. A system for characterizing the condition of a region of 
interest of skin, comprising: 
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a source of illumination of light in at least three spectral bands; 

a camera for acquiring digital images of the region of interest 
based on the light re-emitted from the illuminated region of interest at each of 
the spectral bands, the digital image comprising digital signals whose values 
are a function of the condition of the region of interest; 

memory for storing the digital images provided by the camera; 

a digital processor programmed to perform the steps of: 

segmenting the digital images stored in memory by generating a 
segmentation mask from a digital image in any one of the at least three 
spectral bands; 

estimating at least one value for each digital image at each 
spectral band which is a function of the texture of the portion of the region of 
interest determined by the segmentation mask; 

characterizing the condition of the skin based on the estimated 

values; and 

outputting the characterization of the region of interest. 

45. The system of claim 44, further comprising means for 
suppressing specular reflections from the region of interest. 

46. The system of claim 44, further comprising means for 
calibrating each digital image to provide correction for non-uniformities of 
illumination and response. 

47. The system of claim 44, wherein the digital processor is 
coupled to the source of illumination and to the camera for controlling the 
intensity of illumination and exposure times, respectively. 

48. The system of claim 44, wherein the processor applies the 
segmentation mask derived from the digital images at one spectral band to 
the digital images at the other spectral bands. 

49. The system of the claim 44, wherein the processor 
estimates values separately from digital images at each spectral band based 
on the segmentation mask. 

50. The system of claim 48, wherein the processor compares a 
weighted combination of estimated values against a threshold value. 
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51 . The system of claim 44, wherein the camera records 
monochromatic images and the illumination means comprises: 

a tungsten halogen light source with feedback to stabilize the 
intensity in each wavelength band; 

means for sequentially filtering the light; and 

an optical fiber ring illuminator to distribute the filtered light. 

52. The system of claim 44, further comprising a feedback loop 
for stabilizing the intensity of the light source by the processor. 

53. The system of claim 44, wherein the filter means comprises 
a plurality of interference filters mounted on a wheel for stepping any filter 
into a position intercepting the light from the light source. 

54. The system of claim 44, wherein at least one of the spectral 
bands has a center which is between about 400 to about 500 nanometers, 
and at least one other band centered elsewhere in the visible region. 

55. The system of claim 54, wherein the set of interference 
filters includes a filter whose center lies in at least one spectral band in the 
near infrared range whose center lies between about 750 and 1000 
nanometers. 

56. The system of claim 49, wherein the camera is a single- 
chip, charge-coupled device and the control means comprises a digital 
computer including means for determining exposure times for the camera 
which maximize the signal-to-noise ratio in the image at each spectral band. 

57. The system of claim 44, wherein: 

the source of illumination provides broad-band ("white") light; 

and 

the camera comprises multiple charge-coupled devices which 
detect light in a plurality of spectral bands between the near ultraviolet to 
near infrared. 

58. The system of claim 44, wherein the processor estimates 
values which are statistical measures of local intensity variation in the digital 
images in each spectral band, which are a function of the texture of the 
region of interest. 



WO 98/37811 



44 



PCT/US98/03826 



59. The system of claim 44, wherein the processor estimates 
values based on the ratio of standard deviation of the areas of dermal 
papillae to their mean within the segmentation mask. 

60. The system of claim 44, wherein the processor estimates 
values of the average and standard deviation of the thickness of rete ridges 
within the segmentation masks. 

61 . The system of claim 44, wherein the processor estimates at 
least one value which is a function of the asymmetry of the region of interest 
in each spectral band, for two principal axes of the segmented image by: 

locating the principal axes by computing an orientation angle; 
computing the intensity centroid; 

rotating the digital image such that the principal axes are 
parallel to the image axes; and 

estimating asymmetry values for each principal axis based on 
the intensity centroid; and 

summing the estimated asymmetry values for the two principal 

axes. 

62. The system of claim 44, wherein the processor further 
estimates at least one value which is a function of the blotchiness of the 
region of interest. 

63. The system of claim 44, wherein the processor further 
estimates at least one value which is a function of the irregularity of the 
border of the region of interest by estimating a value which is a statistical 
measure of the deviation of the border of the segmentation mask from the 
border of an ellipse of the same area, aspect ratio, and orientation as the 
segmentation mask. 

64. The system of claim 44, wherein the processor further 
estimates a value which is a function of the gradient at the border of the 
region of interest by estimating a statistical measure of the gradient values of 
the intensity of the digital images across the border of the segmented images, 
at each spectral band. 
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65. The system of claim 44, wherein the processor 
characterizes the type of lesion as invasive or non-invasive. 

66. The system of claim 44, wherein the processor generates 
the segmentation mask from a digital image by: 

removing digital signals from the digital image which correspond 
to hair structures; 

deriving a threshold from a multimodal histogram of intensity 

levels; 

iterattvely applying the threshold to the digital signals of the 
digital image; and 

removing digital signals which correspond to small blob-like 
regions from the masked image. 

67. A system for characterizing the condition of a region of 
interest of skin, comprising: 

a source of illumination of light in at least three spectral bands; 

a camera for acquiring digital images of the region of interest 
based on the light re-emitted from the illuminated region of interest at each of 
the spectral bands, the digital image comprising digital signals whose values 
are a function of the condition of the region of interest; 

a memory for storing the digital images; 

a digital processor including: 

digital processing means for segmenting the digital images 
stored in memory and computing estimated values of parameters which are a 
function of the segmented images; 

digital processing means for automatically characterizing the 
condition of the tissue based on the estimated values; and 

means for outputting the characterization of the region of 

interest. 
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